Workshop on Novel Uses of System Area Networks SAN - 1 Final Program

The emergence of commercially-available network interface controllers (NICs) with remote direct memory access (RDMA) capability and the prospect of their tighter integration with the host memory system motivate the design of distributed systems based on an RDMA paradigm. A recent example is the Direct Access File System (DAFS). DAFS clients communicate requests to servers using lightweight Remote Procedure Calls (RPC) based on low-overhead message passing. Data can be transmitted either in-line with the messages, or via server-initiated RDMA independent of the messages. DAFS clients do not initiate RDMA, despite the potentially lower latency of this mechanism for short transfers. With current NICs, servers that export their entire file cache would have to resort to excessive page wiring to guarantee success of client-initiated RDMA operations. In this paper we present the design of the Optimistic Direct Access File System, a distributed filesystem that enhances DAFS by enabling clients to directly access remote memory pages of cached files exported by servers. Using our proposed NIC support, exported pages are not permanently wired in physical memory except during RDMA operations referencing them. Client RDMA attempts for pages that are no longer memory-resident result in exceptions thrown by the RDMA target (server) and caught by the initiator (client), prompting a switch to an alternative access method. We present simulation results showing that optimistic client-initiated RDMA seldom fails when client working sets fit in server physical memory. Microbenchmarks with a prototype RDMA-capable NIC designed to support the Optimistic Direct Access File System show that client-initiated RDMA can achieve lower file access latency when compared to file access using RPC. The applicability of optimistic client-initiated RDMA extends to other domains, such as Distributed Shared Memory systems.